Global Optimization for Neural Network Training

In this paper, we study various supervised learning methods for training feed-forward neural networks. In general, such learning can be considered as a nonlinear global optimization problem in which the goal is to minimize a nonlinear error function that spans the space of weights using heuristic strategies that look for global optima (in contrast to local optima). We survey various global optimization methods suitable for neural-network learning, and propose the NOVEL method, a novel global optimization method for nonlinear optimization and neural network learning. By combining global and local searches, we show how NOVEL can be used to nd a good local minimum in the error space. Our key idea is to use a user-deened trace that pulls a search out of a local minimum without having to restart it from a new starting point. Using ve benchmark problems, we compare NOVEL against some of the best global optimization algorithms and demonstrate its superior improvement in performance.